Is my meta-analysis useless? Assessment of cumulative research through critical effect size values

Ambra Perugini, Daniël Lakens

What is a critical effect size?

The critical effect size value is the smallest effect that can reach statistical significance given the test, the sample size, and \(\alpha\).

Example: For a two-sided test of a correlation with n = 20, the critical r = 0.44. Effects between −0.44 and +0.44 won’t be significant.

The bigger the sample, the smaller the critical effect size.

Why are they useful?

Before conducting the study:

They quickly tell you what to expect from your research design and test chosen. Instead of crossing fingers you know what will be significant and what will not.

In front of the dead corpse:

They help researchers and reviewers to better contextualize research findings without merely focusing on \(p < 0.05\).

And more..

From a didactic point of view they help comprehend the relationship between statistical significance, sample size and effect size.

Implementation in R

We developed an r package that implements the computation of critical effect size values for correlation, groups comparison, linear regression and meta-analysis.

From sample size (output)
From sample size (code)
From data (output)
From data (code)

$d
[1] -1.077545

$dc
[1] 0.7020467

$bc
[1] 3.273021

$se
[1] 1.601865

$df
[1] 29.65953

$g
[1] -1.050026

$gc
[1] 0.6841176

m1 <- 10
m2 <- 15
sd1 <- 5
sd2 <- 4.25
n1 <- 16
n2 <- 18
critical_t2s(m1 = m1, m2 = m2, sd1 = sd1, sd2 = sd2, n1 = n1, n2 = n2)


    Welch Two Sample t-test

data:  Holz$t01_visperc[Holz$female == 1] and Holz$t01_visperc[Holz$female == 2]
t = 1.4095, df = 298.9, p-value = 0.1597
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.06419229  0.38823118
sample estimates:
mean of x mean of y 
 4.314090  4.152071 

|== Effect Size and Critical Value ==| 
d = 0.1623665 dc = ± 0.2269609 bc = ± 0.2262117 
g = 0.1619587 gc = ± 0.2263909

library(criticalESvalue)
library(psych)
library(psychTools)
data("holzinger.swineford")
Holz <- holzinger.swineford
tt <- t.test(Holz$t01_visperc [ Holz$female == 1], Holz$t01_visperc [ Holz$female == 2])
critical(tt)

The current project

Aims:

Demonstrate the practical utility of critical effect size values.
Investigate the cumulative progress of a research field.

How:

By computing critical effect size values for individual studies within selected meta-analyses, to test whether those studies could have detected a significant effect equal to the meta-analytic estimate.
Assessing the cumulative nature of research my investigating if future studies plan sample sizes based on previous meta-analytic effects.

Meta-analyses requirements

Focus on main effects
Available table of coded studies
Published after the replicability crisis became a thing
Not too recent as well

Case studies

one with large sample size for individual studies
one with small sample size for individual studies

Large Sample Size Case

Shannon et al. (2022)

The meta-analysis investigates the association between social media use and depressive symptoms in young adolescents.

We have calculated critical correlations for the individual studies and compared them to the meta-analytic effect.

In this case, all individual studies had critical effect size values smaller than the meta-analytic effect (100%).
If the meta-analytic effect truly exists, each study can reach significance for it.
In areas with large and easy to collect samples, null results are less likely to stem from insufficient power.

Small Sample Size Case

Talamini et al. (2017)

This meta-analysis investigates differences between musicians and non-musicians in working memory tasks expressing the effect is Hedge’s g due to the small samples.

It has 3 meta-analyses:

long term memory,
short term memory
working memory

We will compute critical values for two tailed t-test, assuming alpha at the .05 threshold and plot them comparing them to the meta-analytic effect.

Long Term Memory

In 0% of cases, the critical value was lower than the meta-analytic effect size, indicating that none of the studies in the meta-analysis could have detected significance for the meta-analytic effect.
When comparing the critical values to the bias corrected meta-analytic effect size the percentage remains 0%, as it cannot go any lower.

Short Term Memory

In 30% of cases, the critical value was lower than the meta-analytic effect size, indicating that 70% of the studies couldn’t reach significance for the meta-analytic effect.
When comparing the critical values to the bias corrected meta-analytic effect size the percentage lowered to 0%.

Working Memory

In 15.79% of cases, the critical value was lower than the meta-analytic effect size, indicating that these studies would have been able to detect a statistically significant effect of that magnitude.
In this meta-analysis no bias was detected.

Who cited the meta did implement knowledge into sample size?

Review of articles citing Talamini et al. (2017)

Of the 214 works citing the meta-analysis, 121 had an accessible manuscript.
Of those 121 manuscripts, 21 had a research design that was compatible with the meta-analysis.
Of those 21, only 2 had run a power analysis but they did not choose the effect size based on that meta-analysis.
A total of 22 manuscripts out of 121 reported a power analysis.

More cases

In other 3 meta-analysis (Adesope et al., 2010; Harkin et al., 2016; Miller et al., 2015) we found that only 34.38% of individual studies had the chance to find a significant result assuming the meta-analytic effect to be true.

One of the studies had a meta-analytic effect of 1.16, if this one is removed the percentage drops to 13.12%.

Harkin et al. (2016): does monitoring goal progress promote goal attainment?

Miller et al. (2015): confidence and judgment accuracy in clinical decision making.

Adesope et al. (2010): cognitive correlates of bilingualism.

Limitations and Future Steps

Selecting arbitrarily 3-4 meta-analyses is not enough to drag conclusions on a field
We will systematically select at least 15 meta-analysis from 2016
We will look into papers citing those meta-analysis as we did for Talamini et al. (2017)

Final Questions to be answered

How many meta-analysis would be enough to drag conclusions on the matter?

What is the percentage of problematic studies that would convince you that the problem is real?

If you are interested in our first paper on Critical Effect Size Values:

Stay tuned for the next one ;)

REFERENCES

Adesope, O. O., Lavin, T., Thompson, T., & Ungerleider, C. (2010). A systematic review and meta-analysis of the cognitive correlates of bilingualism. Review of Educational Research, 80(2), 207–245.

Harkin, B., Webb, T. L., Chang, B. P., Prestwich, A., Conner, M., Kellar, I., Benn, Y., & Sheeran, P. (2016). Does monitoring goal progress promote goal attainment? A meta-analysis of the experimental evidence. Psychological Bulletin, 142(2), 198.

Miller, D. J., Spengler, E. S., & Spengler, P. M. (2015). A meta-analysis of confidence and judgment accuracy in clinical decision making. Journal of Counseling Psychology, 62(4), 553.

Shannon, H., Bush, K., Villeneuve, P. J., Hellemans, K. G., Guimond, S., et al. (2022). Problematic social media use in adolescents and young adults: Systematic review and meta-analysis. JMIR Mental Health, 9(4), e33450.

Talamini, F., Altoè, G., Carretti, B., & Grassi, M. (2017). Musicians have better memory than nonmusicians: A meta-analysis. PloS One, 12(10), e0186773.